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Abstract — The Peer-to-Peer Human-Robot Interaction (P2P- 
HRI) project is developing techniques to improve task coordina- 
tion and collaboration between human and robot partners. Our 
— v/ork-is-raotivated-by the need-to develop effecdve-hiimaa-rGbot 
teams for space mission operations. A central element of our 
approach is creating dialogue and interaction tools that enable 
humans and robots to flexibly support one another. In order to 
understand how this approach can influence task performance, 
we recently conducted a series of tests simulating a lunar 
construction task with a human-robot team. In this paper, we 
describe the tests performed, discuss our initial results, and 
analyze the effect of intervention on task performance. 

I. Introduction 

A. The Peer-to-Peer Human-Robot Interaction Project 
Since February 2004, NASA has been pursuing the “Vi- 
sion for Space Exploration” (VSE) [1]. The primary goal of 
the VSE is to “implement a sustained and affordable human 
and robotic program to explore the solar system and beyond”, 
beginning with robotic missions to the Moon in 2008 and 
leading eventually to human exploration of Mars. To achieve 
this goal, humans and robots will need to collaborate to 
perform basic mission tasks including structural assembly, 
inspection, and maintenance. 

The objective of the “Peer-to-Peer Human-Robot Inter- 
action” (P2P-HRI) project, therefore, is to develop HRI 
techniques that allow humans and robots to work effectively 
together [2]. We use the term “peer-to-peer” not because 
we expect humans and robots to have equal capabilities, 
but to emphasize the idea that humans and robots should 
work as partners, i.e., that their roles should be as balanced 
as possible. In particular, we believe that it is possible to 
compensate for failures and limitations of autonomy through 
interaction and teamwork. 

Our research focuses on making human-robot teams more 
effective for space exploration by: (1) developing natural in- 
teraction mechanisms; (2) reducing the workload associated 
with robot control; and (3) maximizing the work that the 
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humans and robots can accomplish together. In addition, we 
are developing methods that enable human-robot teams to 
operate independently of ground control. This capability is 
needed when communications are intermittent (e.g., explo- 
ration in deep craters) or significantly time-delayed. 

There are three primary components in our approach. 
First, we are developing an interaction framework called 
the “Human-Robot Interaction Operating System” (HRI/OS). 
The HRI/OS is designed to enable humans and robots to en- 
gage in task-oriented dialogue and problem solving. Second, 
we are using computational cognitive architectures to model 
human behavior and make human and robot more under- 
standable to each other. Finally, we are developing evaluation 
procedures using human-robot teams, analog environments, 
and quantitative HRI metrics [3]. 

B. Human-Robot Interaction Operating System 

In order for humans and robots to work effectively to- 
gether, they need to be able to converse about abilities, goals 
and achievements. The Human-Robot Interaction Operating 
System (HRI/OS) is an agent-based system that provides 
coordination and interaction services for human-robot teams 
[4], A key feature of the HRI/OS is that it allows robots to 
ask questions of humans, so that robots can obtain human 
assistance for cognition and perception tasks. 

We designed the HRI/OS to support the performance of 
operational tasks. In space exploration, operational tasks 
include: construction, structural inspection and maintenance, 
and in- situ resource collection and transport [5]. The current 
version of the HRI/OS is implemented using the Open Agent 
Architecture (OAA) [6] and the Internet Communication 
Engine (ICE) middleware [7]. 

The primary components of the HRI/OS (Figure 1) are: 

1 ) Task Manager: The Task Manager (TM) is responsible 
for coordinating and managing the execution of operational 
tasks. It does this by decomposing the overall goal of the 
system into high-level tasks, which are assigned to humans 
or robots for execution. Only a single task is assigned to a 
given agent at a time. Unlike traditional executives, the TM 
does not know anything about low-level task details. Instead, 
it relies on each agent to work in a distributed, independent 
manner, managing and monitoring their own task execution. 

2) Resource Manager: The Resource Manager (RM) is 
responsible for continually tracking which resources (human, 
robots, services) are available and what capabilities each 
resource possesses. When an agent has a request, such as 
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Fig. 1. The Human-Robot Interaction Operating System (HRI/OS) is an agent-based system. 


a task to be performed or a question to be answered, the 
RM generates a prioritfred list of agents to be consulted. 
Unlike facilitation in most agent systems, the RM performs 
delegation using multiple criteria that can vary with time and 
situation, rather than simple match making. In particular, in 
addition to service matching, the RM considers a variety 
of factors including availability, physical location, workload, 
past performance, etc. 

3) Interaction Manager: The Interaction Manager (IM) 
coordinates dialogue between agents. The IM gives each 
agent the ability to communicate with other agents: to ask 
for help, to respond to questions, etc. In the HRI/OS, agents 
communicate with one another via point-to-point “dialogue 
endpoints”. Endpoints allow agents to send a variety of 
data (text, images, sensor readings, etc.) to each other. 
The HRI/OS currently includes graphical and speech user 
interfaces. Other interface modalities (e.g., visual gesturing) 
will be included in the future. 

4) Context Manager: In a complex agent system, keeping 
track of the activities and state of agents over time is a 
difficult task. This is particularly true when multiple agents 
operate in parallel and when activity is observed remotely 
(e.g., via user interfaces). Thus, to facilitate situational 
awareness, we have developed a Context Manager (CM). The 
CM keeps track of everything that occurs while the system 
is running: task status and execution, agent activities, agent 
dialogue, etc. Then, when agents have need to recall history, 
they can query the CM for a summary of information. 


5) Spatial Reasoning Agent: When human-robot teams 
perform operational tasks, such as construction, understand- 
ing and communicating spatial dialogue plays a significant 
role [8]— [10]. In particular, when humans and robots work in 
a shared environment, robots must be able to understand how 
humans perceives space and the relative positions of objects 
around them. Thus, the HRI/OS includes a spatial reasoning 
agent (SRA), which resolves spatial ambiguities in dialogue 
(e.g., “move the light to the left of the box”). The current 
implementation of the SRA is capable of resolving frame 
of reference ambiguities including ego-, addresse-, object-, 
and exo-centric references using a computational cognitive 
model [11]— [13]. 

II. Initial Evaluation 
A. Use Case Study 

In November 2005, to assess our progress in developing 
with peer-to-peer HRI, we conducted a study of a simulated 
construction project involving seam welding and inspection 
by a human-robot team. Seam welding is a task that will be 
required for building and maintaining a variety of structures 
on planetary surfaces [14]. For example, linear welds might 
be used to construct pressure vessels, work hangers, and 
shelters too large to raise into space in one piece [15]. 

The human-robot team consisted of five different members 
working in parallel (Figure 2). In each trial, the team worked 
to retrieve panels from a storage depot, place the panels 
on a structural frame, weld adjacent panels, and inspect the 
welds. Two people wearing spacesuits performed the roles 








Fig. 2. Human-robot team working on a simulated construction project. 
During the study, two astronauts placed panels on a frame, the Robonaut 
humanoid robot created seam welds, and the K10 mobile robot inspected 
the welds. A third person (not shown) remotely supported the team from 
inside a habitat mock-up. 


of astronauts in EVA 1 , retrieving and placing the panels. The 
Robonaut humanoid robot [16] from Johnson Space Center 
worked as a welder using a simulated seam weld process. 
The K10 rover from Ames Research Center was responsible 
for inspecting the welded seams. An additional person, IVA 2 , 
remotely supported the construction project from inside a 
habitat mockup. 

This work scenario provides numerous opportunities for 
dynamic and flexible human-robot interaction. For example, 
a variety of communication acts are possible: human gener- 
ated commands, questions from the robots to the human, etc. 
Additionally, humans may remotely interact with the robots 
(e.g., remotely deploy the inspection robot from a habitat) 
as well as working side-by-side with others (e.g., leading the 
welder robot to a site ready for welding). 


Extravehicular Activity (EVA) is work performed by an astronaut outside 
of a spacecraft 

2 Intravehicular Activity (IVA) is work performed inside a pressurized 
module (spacecraft, habitat, etc. 


B. Data Collection and Baseline 

During the study, we made extensive use of video and 
audio to record human-robot interaction. Three observers 
used logging software to code human and robot activity. For 
each session, we logged time spent moving, time on task, and 
intervention time (the time a human or robot spent helping 
another agent). From this data we are able to determine the 
free time that any of the agents had. 

To establish baselines, we collected the time required for 
each agent to perform primitive tasks: retrieve and assemble 
panels (astronauts only), move between panels (all), seam 
weld (astronaut and Robonaut), inspect weld (K10 only). 
Table I shows the baseline data for the EVA astronauts, K10, 
midJRohonaut. This baseline data ca n be used to construct 
time-based workflows of different team configurations. 


TABLE I 
Baseline data 


Agent 

Primitive task 

Mean time (sec) 


move between panels 

3.9 

EVA astronauts 

weld seam 

16.5 

retrieve panel 

13.5 


assemble panel 

7.7 - 

K10 

inspect seam 

11.3 

move between panels 

12.6 

Robonaut 

weld seam 

move between panels 

66.1 

27.4 


We should note that it is unlikely that human-robot teams 
will be able to work as quickly as comparably sized human 
teams any time soon. But, because cost pressures and other 
constraints (e.g. payload launch capacity) will keep astronaut 
teams small, exploration missions will need to include teams 
of humans and robots. Thus, we are most concerned with 
identifying task workflows that maximize human-robot team 
productivity, while reducing the number and duration of 
EVA’s. 

C. Preliminary Results 

One series of tests included seven trials, in which six 
panels were placed, welded, and inspected by the human- 
robot team. The start condition and work sequence in each 
trial was identical. During these trials, the two EVA astro- 
nauts commanded and provided assistance to the robots. The 
IVA astronaut communicated with the EVA astronauts. IVA 
also teleoperated and assisted K10 with inspections when 
needed. Trials 2 to 7 were successfully completed. Figure 3 
shows a timeline from Trial 6, broken down into four activity 
categories: free time, time on task, time on task with help, 
and intervention (providing help). 

To better understand how the EVA astronauts worked, we 
computed the percentage of time the astronauts dedicated to 
each activity category during each trial (Figure 4). The as- 
tronauts spent about the same percentage of time on the task 
during all runs. They spent little time intervening (providing 
help to robots). Overall, the EVAs had a high percentage of 
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Fig. 3. Activity timeline of human-robot seam welding. 


free time, most of which occurred while waiting for robots 
to finish working at the end of each trial and which would 
be available for other work. 
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Fig. 4. EVA astronauts time usage 
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For the IVA we coded four work tasks: monitoring (user 
interface or line-of-site observation), robot teleoperation, - 
intervention (providing help to robots), and human commu- 
nication. During all trials, we also asked the IVA to perform 
a secondary task (reading a newspaper) to better simulate 
actual work (i.e., so that his attention would not be focused 
only on the team). Figure 5 shows the percentage of time 
the IVA spent in each of these activities. 



trial 


m intervention 
0 teleoperation 
0 second task 
Mi monitoring 
m human comm 


Fig. 5. IVA astronaut time usage 


Fig. 6. Effect of interventions on EVA timeline: (a) no interventions, (b) 
intervention while human and robot working together, (c) intervention after 
human has started other work. 


By far the greatest percentage of the TVA’s time was spent 
on communication. This reflects a need for having the status 
of the robots and TM more visible to the EVAs. In Trial 7, 
the IVA was teleoperating K10 while communicating with 
EVAs but this was not the norm. 

III. Intervention and EVA 

To better understand how human intervention (i.e., an as- 
tronaut assisting a robot) effects EVA, consider the following 
two measures of EVA productivity: 

1) The amount of useful work that astronauts can do. 

2) The amount of time astronauts must spend in EVA. 

Depending on the nature and the context of the task, it may 

be important to maximize the first measure (useful work). For 
other tasks, minimizing the second (exposure) may be the 
priority. The ideal, of course, is a system that can maximize 
useful work while minimizing exposure. 

Interventions reduce the amount of time the astronauts 
have available for other work in EVA. Because of the time 
associated with egress and regress, if robots are “unreliable” 
and require many interventions, the astronauts may need 
to stay in EVA and “wait” to handle for these unplanned 
interventions. Even if the astronauts were to remain out on 
EVA trying to get other work done, their work would be 
constantly interrupted by robots requesting intervention. This 
would directly reduce the productivity of the human-robot 
team. 

Figure 6 illustrates the effects of interventions on different 
parts of an EVA timeline. A nominal timeline in which the 
human-robot system performs a collaborative task without 
unplanned interventions is shown in Figure 6a. The maxi- 
mum EVA time, Tbva > is the maximum amount of time that 
the human-robot system has to perform a primary task (Task 
1). The nominal amount of time for the human-robot system 
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to perform a specific task is labeled T tas k- The nominal 
amount of time required for the astronauts to perform then- 
part of Task 1 is labeled T tas k BVA - The time remaining once 
the astronauts complete Task 1 is the time available for them 
to perform other work. 

Interventions that occur while the astronauts are still per- 
forming their part of Task 1 increase T ta sk BVA and T task as 
shown in Figure 6b. Unplanned interventions after T taskBVA 
lead to a situation in which the astronauts may have to remain 
in EVA in order to intervene. If the astronauts do not remain 
in EVA, time penalties associated with egress/regress may 
be incurred. This situation is shown in Figure 6c. Each of 
these events significantly increase exposures and decreases 
the time available for other work. 

If, however, the human-robot system is very “reliable” 
(i.e., robots do not often need help), then astronauts have 
two choices. On the one hand, when they finish their primary 
task, they can stay in EVA and begin working on other tasks. 
This increases the amount of useful work. Alternatively, 
the astronauts can assume that the robots will complete 
their tasks without intervention, and thus they can leave 
EVA. This helps minimize exposure. In other words, a 
very “reliable” human-robot system provides the option to 
maximize different productivity measures. 

We can examine the effect of intervention on EVA using 
an analysis inspired by classic reliability techniques [11]. To 
begin, we first define the following terms: 

• Mean Time Between Interventions (MTBI) is the mean 
time that a human-robot system operates nominally (i.e., 
outside of intervention). MTBI is analogous to Mean 
Time Between Failures (MTBF) [18]. 

• Mean Time to Intervene (MTTI) is the mean time 
humans and robots spend in intervention. MTTI is 
analogous to Mean Time To Repair (MTTR) [18]. 

MTBI is indicative of how often intervention is needed 
(e.g., how frequently robots ask for help). MTBI is a func- 
tion of several factors: the environment (and uncertainty in 
the environment) the system or robot is operating in; the 
autonomy of the system or robot; and inherent component 
or system reliabilities. 

MTTI is indicative of intervention time (e.g., how long it 
takes to answer a robot’s question). MTU is a function of 
many variables, including (but not limited to): the nature of 
the problem requiring intervention; the amount of informa- 
tion a robot or astronaut is able to gather about the problem; 
the amount and type of information that can be transferred 
from robot to human; the physical distance between human 
and robot; the lag in communications between human and 
robot; and available resources. 

A. Time Available for Other Work 

The lifetime of a system is usually considered as having 
three phases: (1) infant mortality with decreasing failure 
rate, (2) normal life and (3) wear-out with an increasing 
failure rate. During normal life, the conditional probability 
of failure. A, given that a component has survived to a given 
time t is generally constant. 
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Fig. 7. Time for other work as a function of MTBI and MTTI. EVA values 
used: Teva = 8 hours, T taskBVA = 50 min, T task = 100 min. 

To examine the influence of MTBI and MTTI on EVA, 
we will focus on the normal life phase. Assuming that A is 
constant (i.e. that interventions will occur at a constant rate): 

MTBI = i = constant 
A 

we can compute the time available for other work using: 

T’other = T EV A ~ Tt a sk EVA ~ MTTI 

where T ot her is the “time for other work”, Teva is the 
total EVA time, T taskEVA is the nominal time required for 
astronauts to perform their part of the task, and T tas k is the 
nominal time for the human-robot team to complete the task. 

Figure 7 shows how time for other work depends on 
MTBI, assuming constant MTTI and representative EVA val- 
ues. Increases in MTTI reduce the time available for human 
agents to do other work. However as MTBI is increased, 
the sensitivity to MTTI quickly decreases. Also, increases in 
MTBI result in increased time available for other work, and 
the sensitivity to MTBI increases as MTTI increases. This 
suggests that a designer may be able to compensate for large, 
or uncertain, MTTI and increase time available to do other 
work by modestly increasing MTBI. 

B. Intervention Probability 

To examine how MTBI and MTTI influence the proba- 
bility that astronauts will need to intervene, we model the 
occurrence of interventions using a Poisson distribution. The 
probability, F(t), that at least one intervention will occur 
between when the astronauts finish their part of the task and 
when the human-robot system finishes the task is then: 

r't'task.tot 

F(t) — — - 1 exp(— / h(s ) ds) 

” Tta.sk E y A ,tot 

where T ta sk E vA,tot is the time required for the astronauts to 
perform their part of the task (including interventions) and 
Ttask, tot is the total time to perform the task. 

Figure 8 shows the probability that intervention is required 
(after the astronauts have finished their part of the primary 




together, supporting one another regardless of spatial distri- 
bution, communication channel, and user interface. 

During the next few years, our goal will be to apply 
peer-to-peer HRI directly to a variety of exploration mission 
systems, including the Crew Exploration Vehicle (CEV) and 
lunar surface systems (landers and rovers). In addition, we 
plan to study how peer-to-peer HRI can be applied to a 
wide range of in-space and surface exploration activities 
including structural assembly, inspection, engineering survey 
and resource mapping, payload transport, site preparation, 
and infrastructure deployment. 
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Fig. 8. Probability of Intervention as a function of MTBI and MTTI. EVA 
values used: Teva = 8"hours, T tas k EVA = 50 min, T tas k = TOO min. 

task) as a function of MTBI, assuming constant MTTI and 
representative EVA values. Increasing MTTI by an order 
of magnitude only shifts the curve slightly. Furthermore, 
as MTBI increases, the effect of changes in MTTI or 
MTBI decreases. This suggests that uncertain MTTI may 
not significantly influence the probability of intervention. 

IV. Future Work 

The use case study revealed a number of deficiencies with 
the current system. In particular, there were a number of 
times when astronauts used incorrect dialogue. Because the 
current speech interface does not provide confirmation, the 
astronauts did not know if a command was received or if 
the grammar was incorrect. Thus, both the dialogue and the 
speech interface need to be made more flexible and robust. 

We also observed that the astronauts sometimes forgot 
to inform the TM when they had completed their work. 
Although the TM tracks the progress of robot agents, a 
similar facility is needed for tracking humans in their work. 
This will require more extensive cognitive modeling and the 
use of activity monitoring techniques. In addition, our long- 
term goal is to extend the HRI/OS to support large human- 
robot teams. To do this, the TM will need to reason about, 
and plan for, resource usage. 

By far the biggest issue, however, is that of status monitor- 
ing. At present, it is difficult for EVA astronauts to assess the 
status of robots (progress, health, etc.) and software agents. 
This deficiency impacts both situation awareness and context 
switching. Although it is possible for the IVA astronaut to 
provide status reports as an overseer, frequent human-human 
communications make it difficult for the IVA to accomplish 
other tasks. Thus, we need to develop status monitoring 
mechanisms that can function independent of the IVA. 

V. Conclusion 

The tools and techniques that the Peer-to-Peer Human- 
Robot Interaction project is developing have significant po- 
tential for improving the human-robot teams. In particular, 
we believe that software frameworks such as the HRI/OS 
will enable humans and robots to work more productively 
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